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r\l Abstract. We consider two core algorithmic problems for probabilistic 

, , verification: the maximal end-component decomposition and the almost- 

C^ sure reachability set computation for Markov decision processes (MDPs). 

^TJ For MDPs with treewidth k, we present two improved static algorithms 

^^ for both the problems that run in time 0{n ■ k'^'^^ ■ 2*^) and 0{m ■ logn • 

(^ k), respectively, where n is the number of states and m is the number 

CO of edges, significantly improving the previous known 0{n ■ k ■ Vn ■ k) 

bound for low treewidth. We also present decremental algorithms for 
j.^ both problems for MDPs with constant treewidth that run in amortized 

Q logarithmic time, which is a huge improvement over the previously known 

algorithms that require amortized linear time. 



1 Introduction 



^ In this work we will present efficient static and decremental algorithms for two 

^" core graph algorithmic problems in probabilistic verification when the graph has 

i-I low treewidth. We start with the basic description of the model, the problem, 

f~^ and its importance. 

.^b Markov decision processes with parity objectives. The standard model 

f^ of systems in probabilistic verification that exhibit both probabilistic and non- 

CO deterministic behavior are Markov decision processes (MDPs) [19] . MDPs have 

^^ been used for control problems for stochastic systems |17j . where nondetermin- 

^ ism represents the freedom of the controller to choose a control action, and 

the probabilistic component of the behavior describes the system response to 
, ; control actions; as well as in many other applications |12I2I18| . A specification 

c3 describes the set of good behaviors of the system. In the verification and con- 

trol of stochastic systems the specification is typically an w-regular set of paths. 
The class of w-regular languages extends classical regular languages to infinite 
strings, and provides a robust specification language to express all commonly 
used specifications, such as safety, liveness, fairness, etc. 26J. A canonical way 
to define such w-regular specifications are parity objectives. Hence MDPs with 
parity objectives provide the mathematical framework to study problems such 
as the verification and control of stochastic systems. 

The analysis problems. There are two types of analysis for MDPs with par- 
ity objectives. The qualitative analysis problem given an MDP with a parity 



objective, asks for the computation of the set of states from where the parity 
objective can be ensured with probability 1 (almost-sure winning). The more 
general quantitative analysis asks for the computation of the maximal probabil- 
ity at each state with which the controller can satisfy the parity objective. 

Significance of qualitative analysis. The qualitative analysis of MDPs is an 
important problem in verification. In several applications the controller must 
ensure that the correct behavior arises with probability 1. For example, in anal- 
ysis of randomized embedded schedulers, the relevant questions is whether every 
thread progresses with probability 1 [14j . Moreover, even in applications where 
it is sufficient to satisfy the specification with probability p < 1, the correct 
choice of p is a challenging problem, due to the simplifications introduced dur- 
ing modeling; for example, for randomized distributed algorithms it is common 
to require correctness with probability 1 (see, e.g., |23I21I25J ). Furthermore, in 
contrast to quantitative analysis, qualitative analysis is robust to numerical per- 
turbations and precise transition probabilities, and consequently the algorithms 
for qualitative analysis are discrete and combinatorial. Finally, the best known 
algorithms for quantitative analysis of MDPs with parity objectives first per- 
form the qualitative analysis, and then a quantitative analysis on the result of 
the qualitative analysis |12|13|10] . 

Core algorithmic problems. The qualitative analysis of MDPs with par- 
ity objectives relies on two graph algorithmic problems: (1) the maximal end- 
component decomposition; and (2) the almost-sure reachability set computation. 
An end-component C in an MDP is a set of states that is strongly connected 
and closed (no probabilistic transition from C leaves C), and a maximal end- 
component is an end-component which is maximal with respect to inclusion or- 
dering. The maximal end-component (MEC) problem generalizes the sec (maxi- 
mal strongly connected component) decomposition problem for directed graphs, 
and recurrent classes for Markov chains. The almost-sure reachability set for a 
set U of target vertices is the set of states such that it can be ensured that the set 
U is reached with probability 1 (in other words, it is the qualitative analysis for 
reachability objectives). The qualitative analysis problem for MDPs with parity 
objectives with d-priorities can be solved with logd calls to the MEC decompo- 
sition problem and one call to the almost-sure reachability problem |6] . Thus the 
MEC decomposition and the almost-sure reachability set computation are the 
core algorithmic problems required for the qualitative analysis of MDPs with 
parity objectives. In addition to qualitative analysis of MDPs with parity ob- 
jectives, several algorithms for quantitative analysis of MDPs with quantitative 
objectives such as limsup and liminf objectives 18 , combination of mean-payoff 
and parity objectives [S], and multi-objective mean-payoff objectives 5 , rely 
crucially on the MEC decomposition problem. 

Dynamic algorithms. In the design and analysis of probabilistic systems it 
is natural that the systems under verification are developed incrementally by 
adding choices or removing choices for player 1, whereas the probabilistic choices 
which represent choice of nature or uncertainty remain unchanged. Hence there 
is a clear motivation to obtain dynamic algorithms for MEC decomposition and 



almost-sure reachability set for MDPs that achieve a better running time than 
recomputation from scratch when player-1 edges are inserted or deleted. 

Previous results. The current best known algorithms for both the MEC de- 
composition and the almost-sure reachability set computation require 0{m ■ 
min(-y/m, n^'^)) time [6l7j . where n is the number of states and m is the number 
of transitions (edges) . Using a well-known fact that graphs of treewidth k have 
0{n-k) edges, one can obtain 0{n-k-Vn ■ k) algorithms for MEC decomposition 
and almost-sure reachability set computation (they follow directly from the gen- 
eral 0{m ■ y/rn)-tune algorithm). The best known incremental and decremental 
algorithms for both problems require amortized linear time {0(n) time) [B]. 

Our contributions. In this work we consider MDPs with low treewidth. The 
concept of treewidth and tree decomposition of graphs was introduced in |24j . On 
one hand treewidth is a very relevant graph theoretic notion that measures how a 
graph can be decomposed into a tree, on the other hand, most systems developed 
in practice have low treewidth. For example, it has been shown that the control 
flow graphs of goto free Pascal programs have treewidth at most 3, and that the 
control flow graphs of goto free C programs have treewidth at most 6 [27j . It was 
also shown in 27J that tree decompositions, which are very costly to compute in 
general, can be generated in linear time with small constants for these control 
flow graphs. Our main results are efficient static and decremental algorithms 
for the MEC decomposition and the almost-sure reachability set computation 
for MDPs with low treewidth. Several benchmarks in PRISM are probabilistic 
programs written in programming languages mentioned above and consequently 
have small treewidth, and our results are relevant for such MDPs. The details 
of our contribution are as follows: 

1. We present two improved static algorithms both for the MEC decomposition 
and the almost-sure reachability set computation for MDPs with treewidth 
k that run in time 0{n ■ k^-^^ ■ 2'^) and 0{m ■ logn • k), respectively, where 
n is the number of states and m is the number of edges (also note that for 
treewidth k we have m = 0{n ■ k)). For MDPs with low treewidth, our new 
linear-time algorithms are significant improvements over the previous known 
0{n ■ k ■ \/n ■ k) algorithms for both the problems. 

2. We present decremental algorithms for the MEC decomposition and the 
almost-sure reachability set computation for MDPs with treewidth k that 
require O(fc-logn) amortized time, which is a huge improvement for constant 
treewidth over the previous algorithms that require 0{n) amortized time. 

Our key technical contribution is as follows: for MDPs we establish a separation 
property for the almost-sure reachability set that allows us to use tree decompo- 
sition to obtain the 0{n ■ k"^-^^ ■ 2'^)-time static algorithm. A similar intuition also 
works for the MEC decomposition problem. We then view the MEC decomposi- 
tion and the almost-sure reachability set computation problems as decremental 
graph problems, and use dynamic graph algorithmic techniques to obtain the 
0(m- log n-fc)-time static algorithms and the decremental algorithms. Note that 
when edges are inserted, the treewidth of the graph may increase and the tree 



decomposition can change. Thus, incremental algorithms with polylogarithmic 
amortized cost remain an interesting open question (even for sec decomposition). 

Related works. The notion of treewidth is studied in context of many graph 
theoretic algorithms, see [3] for an excellent survey. In verification, the prob- 
lem of low and medium treewidth has been considered for efficient algorithms 
for parity games: a polynomial time algorithm for parity games with constant 
treewidth was presented in |22j ; a recent improved result for constant treewidth 
was presented in |16j ; and the algorithmic problem of parity games with medium 
treewidth was considered in |15) . Though the games problem has been studied 
with the treewidth restriction, to the best of our knowledge, improved algorithms 
for MDPs have not been considered with the treewidth restriction. 

2 Preliminaries 

In this section we first present the basic graph theoretic definitions of the MEC 
decomposition and the almost-sure reachability set computation, and then define 
the notions of treewidth. 

2.1 MEC decomposition and almost-sure reachability 

Markov decision processes (MDPs). A Markov decision process (MDP) G = 
((y,-E), (Vi, Vp), (5) consists of a finite directed MDP graph (V,i?), a partition 
(Vi,yp) of the finite set V of vertices, and a probabilistic transition function 
5: Vp — )■ Viy), where 'D^V) denotes the set of probability distributions over 
the vertex set V , such that for all vertices u G Vp and v ^ V we have uv G 
E iff 5{u){v) > 0. An edge uv G E is a. player-1 edge if m G V\. For the 
algorithmic problems we will consider, the probabilistic transition function will 
not be relevant and we will consider the MDP graph along with the partition. 

Maximal end-component decomposition. For the maximal end-component de- 
composition, the input is a directed graph G = {V, E) and a partition (Vi, Vp) 
of its vertex set (i.e., the MDP graph and the partition). An end-component 
/7 is a set of vertices such that the subgraph induced by U is strongly con- 
nected and for each edge uv G E, \i u G U C^Vp then v G U. If t/i and U2 are 
two end-components and [/i n C/2 7^ 0, then Ui U U2 is also an end-component. 
The maximal end-component (MEC) decomposition consists of all the maximal 
end-components of V and all vertices of V that do not belong to any MEC. 

Almost-sure reachability. For almost-sure reachability, the input is an MDP and 
a target set U <^V of vertices, and the goal is to compute the set A of vertices, 
such that player 1 can ensure that the set U is reached with probability 1. We 
first note that given the target set U, we can add a new vertex s as the new 
target vertex, and transform the set U such that all out-edges from vertices in 
U end up in s, and the vertex s has only a self-loop. Thus we will consider the 
case when the target set is a single vertex s. We first reduce the computation of 



the almost-sure reachability set for a target vertex s to the following problem. 
The input is a directed graph G — {V,E), a partition (Vi, Vp) of its vertex set 
(the MDP graph and the partition), and a target vertex s G V. The goal is to 
compute a maximal (w.r.t inclusion) subset Q '^V, such that the following two 
conditions are satisfied: 

— for every q € Q, there exists a path from q to s consisting only of vertices in 
Q (global condition), and 

— for every uv G E, ii u G Q DVp, then v E Q (local condition). 

First observe that if Qi C y and Q2 '^ V both satisfy the global and the local 
conditions, then so does QiU Q2- It follows that there is a unique maximum set 
A* CV that satisfies both the global and the local conditions. The resulting set 
A* is the almost-sure reachability set (in the following also called an ASR set). 
Let A be the almost-sure reachability set and A* be the largest set that satisfies 
the two conditions (the global and the local conditions). 

Lemma 1. We have A — A* . 

Since A = A* we consider the graph theoretic problem of computation of A* 
(i.e., the largest set satisfying the global and the local conditions). 

Notations. Let G be a directed graph. We denote its vertex and edge set by 
V{G) and E{G), respectively. By G[S] we denote the subgraph of G induced on 
vertices belonging to S, whereas by G \ S' we denote the subgraph of G induced 
on V{G) \ S. A separator is a subset S C V{G), such that G \ S has more 
connected components than G (when all edges are treated as undirected). 

2.2 Tree decomposition of graphs 

We begin by introducing some definitions depicted in Fig. Ill 

Definition 1. Let G = {V,E) be an undirected graph. A tree decomposition 
of G is a pair {B, T), where B is a family _Bi, . . . , _B„ of subsets of V (called 
bags) and T is a tree, whose nodes are sets Bi. The decomposition satisfies the 
following properties: 

1. [_}Bi = V (bags cover vertices). 

2. For every uv E E there exists Bj, such that u,v G Bj (bags cover edges). 

3. For every v (zV the sets Bi containing v form a connected subtree of T . 

Definition 2. The width of a tree decomposition (i?, T) is equal to 
maxSiSB \Bi\ ~ 1. The treewidth of an undirected graph is the minimal possi- 
ble width of its tree decomposition. 

The concept of treewidth grasps the sparseness of a graph. Treewidth of a 
tree is equal to 1, while cliques on n vertices have treewitdth n — 1. Note that 
the definitions are given for undirected graphs, but they can also be applied to 
directed graphs. In such case, we treat all edges as undirected. 
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Fig. 1. A sample graph (left), its tree decomposition (center, edges covered by 
each bag have been marked for illustration) and a nice tree decomposition (right). 

Definition 3. A tree decomposition {B, T) is called nice if T is a rooted tree 
and each of its nodes Bi belongs to one of the following four types: 

1 . leaf — Bi is a leaf of T and \Bi\ — 1 . 

2. introduce — Bi has a single child Bj and Bi = Bj U {v}. 

3. forget — Bi has a single child Bj and Bi = Bj \ {v}. 

4- join — Bi has two children Bj and Bk, and Bi = Bj = Bk- 

Theorem 1 ([3]). Let G be a graph of treewidth k. Assuming that k is a con- 
stant, the tree decomposition of G of width k can be computed in 0{n) time. 

Lemma 2 (see e.g. [20j). A tree decomposition can be transformed, in linear 
time, into a nice tree decomposition of the same width, consisting of 0{n) nodes. 

We also use the following well-known fact, which can be derived from the 
definition. Informally, consider a vertex ts of a tree decomposition T of a graph 
G. Assume that it contains a bag B C V{G). Denote the connected components 
of T\{tB} by Ti, . . . , Tfc. Then, the trees Ti correspond to connected components 
of G\B, namely bags from each Ti cover vertices from one connected component. 

Lemma 3. Let B be a bag in a node ts of the tree decomposition of G. Consider 
the connected components Ti, . . . , Tfc of T\ {is}. Then the following hold: 

1. Either B is a separator in G or all but one Ti consist solely of bags that are 
subsets of B. 

2. Each path from a vertex u ^ B covered with a bag in Ti to a vertex v ^ B 
covered with a bag in Tj (i ^ j) goes through a vertex in B. 

Observe that a vertex not belonging to B can be covered by bags from at 
most one Ti. This is because the set of bags covering a given vertex forms a 
connected subgraph of T. 



6 



3 Algorithms for MDPs with Constant Tree-width 

In this section we will first present an algorithm for computing the ASR set, 
whose running time depends linearly on the size of the input MDP graph, where 
the input graph has constant treewidth. We will then present the linear-time 
algorithm for MEC decomposition for MDPs with constant treewidth graphs. 
The algorithms require that a tree decomposition of the graph of width k is given 
and run in time that is exponential in k. If fc is a constant, the decomposition 
can be computed in linear time (see Theorem [I]) . To simplify presentation, we 
use Lemma [2] to transform the decomposition to a nice one. 

3.1 Almost-sure reachability 

Our algorithm for the ASR set computation is based on the following separation 
property. 

Lemma 4. Let B be a subset ofV{G), such that the target vertex s belongs to B. 
Denote the connected components of G\B by Ci, . . . ,Ck- Assume that we know 
the intersection of the ASR set A with B. For each i — 1, . . . , fc, construct the 
subgraph of G induced on CiUB. Add to this graph a set of edges {vs\v £ ACiB}, 
thus obtaining a patched component Ci. Denote by Ai the ASR set in Ci. Then 
we have A = AiU . . .U Ak- 

Lemma |4] says that if we know AO B, then we can compute the ASR set 
independently in each (patched) connected component oi G\B and then simply 
merge the results. Since we assume that G has low treewidth, it also has sep- 
arators of small size. Thus, in the algorithm we can guess An B, by checking 
all possibilities. We do not prove the separation property explicitly. Instead, we 
give the algorithm inspired with this property and then prove its correctness. 
The property will follow from Lemma [61 Let us now describe the details. 

Denote the nice tree decomposition of G by T. We add the target vertex s 
to every bag of T. Note that this might increase the width of the decomposition 
by at most one, but the modified T is still a valid tree decomposition. However, 
T is no longer a nice decomposition, as the leaf nodes now contain two vertices. 
We fix this, by adding a child {s} to every leaf. Then we choose an arbitrary 
leaf as the root. 

The algorithm is based on a bottom-up dynamic programming on T. Fix a 
node d of T, and assume that it contains a bag i?^- Denote by Gj, the subgraph 
of G induced on the vertices enclosed in the bags from the subtree rooted at d. 
By Lemma Isj Bd separates Gd \ Bd from the rest of the graph. 

Now, according to Lemma [ij for each subset B' C Bd we should add edges 
{vs\v € B'} to Gd and compute the ASR set of the obtained graph. However, 
we do a slightly different thing: instead of adding edges, we just treat all vertices 
of B' as target vertices (note that this has the same effect as adding edges from 
vertices in B' to s). This motivates the following definition of a partial solution. 
Partial solution is defined with respect to a subgraph of Gd ^ G, and, informally, 
it is the set of vertices from Gd that will be included in the ASR set. 



Definition 4. A partial solution for a node d is a subset of V{Gd)- A partial 
solution P is called valid, if the following hold. 

i. For every v ^ P OVp and every edge vu £ E{Gd), we have u £ P. 
ii. For every v £ P there exists a path in P that connects v to some vertex in 

PnBd. 

We denote by P{B',d) the maximal (w.r.t. inclusion) valid partial solution 
(for node d) which satisfies P{B' ,d) f] Bd = S'P] Observe that the definition 
is unambiguous, since the union of two valid partial solutions is a valid partial 
solution. However, it might be the case that for some choice of B' there are no 
feasible valid partial solutions. In such a case we set P{B',d) = _L. We later 
show that ii B' = An Bd, then P{B', d) = Ar\ V{Gd)- 

The algorithm considers possible ways of including a subset of Bd in the 
ASR set, by iterating through all valid subsets B' C Bd- A subset B' C Bd 
is valid, if it contains the target s and for each v E B' D Vp and every edge 
vu € ED {Bd X Bd), we have u S B'. In particular, for any valid partial solution 
P containing s, the set P Bd is a valid subset. 

In addition to P{B',d), for each valid B' C B and each pair of vertices 
x,y € B' , we compute whether there exists an x-to-y path consisting of vertices 
contained in P{B' , d). Formally, we compute the transitive closure oiG[P{B' , d)], 
restricted to B' . In the following this transitive closure is denoted by TC{B', d). 
Note that it is a subset of Bd x Bd- 

The algorithm is run bottom-up on T. For a given node d and each valid sub- 
set B' it computes P{B' , d) and TG{B' , d), using the values from the children of 
d. There are four cases to consider, one for each type of node. In the description, 
we assume that the value _L is propagating. This means, that the result of any 
set operation involving _L is _L. 

— Leaf The bag contains a single vertex s (the target), so the transitive closure 
is empty and we set P{{s}, d) = {s}. 

— Join Denote the children of d by ci and C2. In this case, we set P{B' ,d) = 
P{B', ci) U P{B' , C2), so the transitive closures from the children have to be 
combined, i.e. TC{B',d) = (TC(J5',ci) UTC(B', ca))*. The asterisk denotes 
the operation of computing the transitive closure. 

— Introduce Denote the introduced vertex by w and the child of d by c. For 
all valid subsets B' C Bd that do not contain w, we set P{B' , d) = P(B', c) 
and TC{B', d) = TG{B', c).Iiw€ B\ then P{B', d) = P{B' \ {w}, c) U {w}. 
Thus, to compute the transitive closure in this case, we take TC{B' \ {vu}, c), 
add all edges incident to w and compute the transitive closure of the obtained 
set. Hence, TC{B',d) = {TC{B' \ {w},c) U {wz G E{G)\z e B'} U {zw G 
E{G)\z&B'})*. 

— Forget Denote the vertex that is forgotten by w and the child of d by c. 
Hence, the bag in the child Be is equal to Bd U {w}. We check whether we 
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In the end we prove slightly less about the values P{-, •) that are computed by the 
algorithm, but it is convenient to think about them this way. 



can include w in P{B',d). For this, condition (ii) (of Definition E| has to 
hold, i.e., there has to be a path in P{B', d) that connects w to some vertex 
in B' . We claim that it suffices to check, whether w has any out-edges in 
TC{B' U {w},c). If this is the case, then w is connected to some vertex 
from B' in P{B' U {w},c), so P{B',d) = P{B' U {w},c) and we can set 
TC[B',d) =rC(B'U{w},c)n(B'xB'). Otherwise, we just copy the result 
from the child, that is set P{B',d) = P{B' ,c) and TC{B',d) = TC{B',c). 

Finally, the ASR set computed by the algorithm is stored in P{{s},r). We 
now prove the correctness of the algorithm with the following two lemmas. The 
proof of Lemma [5] is presented in the appendix. 

Lemma 5. For each node d and each valid subset B' C Bd, if P{B' , d) ^ _L, 
then P{B',d) is a valid partial solution and TC{B',d) is computed correctly. 

Lemma 6. Let A be the maximum ASR set. For each node d, P{A n Bd, d) = 

AC^V{Gd). 

Proof. The proof proceeds by induction on the depth of the subtree rooted in d. 
First, it is easy to see that Ar\Bdis a. valid subset for d. Moreover, A{^V{Gd) is 
a valid partial solution for d. Let us check condition (ii) of Definition [4] For each 



V €: A there exists an u-to-s path p in A. Denote by u; the last vertex of p that 
lies inside Ar\V{Gd)- By Lemma l3J vi € Bd and consequently also vi e Af] Bd- 

— Leaf P{A n Bd, d) - P({s}, d) - {.s} = An V{Gd). 

— Join By induction hypothesis we have P{A n Bc-,Ci) = A O V{Gci), for 
i = 1, 2. From the definition P{A n Bd, d) = P{A n Bd, ci) U P{A n Bd, C2) = 

PiAnB,„ci)uPiAnBc„c2) = (Ani/(G,j)u(^ny(GcJ) = An(y(GcJu 

V{Gc,))^AnViGd). 

— Introduce If A does not contain the introduced vertex w, then P{A O 
Bd, d) = PiA nB„c) ^ An V{Gc) ^ An iV{Gd) \ {w}) ^ An ViGd). 
otherwise, if w e A we have P{A n Bd, d) = P{{A n Bd) \ {w}, c) U {w} = 

{A n v{G,)) u M - A n {v{Gd) \ {w}) \j{w}^An v{Gd). 

— Forget Denote the forgotten vertex by w. 

We claim that w € A ]S A n B^ \s a. valid subset of Be and w has some 
out-edges in TG{A n Bc,c). (^) It follows immediately that An Be \s a, 
valid subset. Moreover, since there is a path from w to s in A, by Lemma[3j 
there has to be a path that connects w to some vertex in {A n Be) \ {w} in 
P{AnBc,c). (^) Assume that w ^ A. We show that AuP((AnBc)U{'u;}, c) 
is an almost-sure reachable set that is larger than A. Indeed, we know that 
from every vertex in P{{A n Be) U {w},c) there is a path to a vertex in 
{A n Be) U {w}, hence also a path to A n Be. In addition, from every vertex 



in An Be there is a path to s. It follows easily that condition (ii) of being 
an ASR set also holds, which shows the desired. 

Now, ii w & A, the algorithm sets P{A n Bd, d) = P{{A n Bd) U {w}, c) = 
P{A nBe,c) = An V{Ge) ^ An V{Gd). On the other hand, ii w ^ A, we 
have P{A n Bd, d) = P{A n Bd, c) = P{A n {Be \ {w}),c) = P{A n Be, c) = 

AnviGe) = Anv{Gd). □ 



By applying Leniina|6]to the root r of the tree decomposition, we obtain that 
P{A n V{G), r) = An V{G) ~ A. Let us now analyze the running time. 

Running time analysis. We represent TC{-, •) with a (/c + 2) x (fc + 2) matrix. 
(In the original tree decomposition bags had size A: + 1, but then we added 
the vertex s to every bag.) The sets P{-,-) can be represented implicitly, that 
is for a set P{B,d) we store how it can be obtained from the respective sets 
contained in the children of d. This requires constant memory for each set. We 
iterate through 0(2'^) subsets of each bag. Checking whether a set is valid boils 
down to inspecting all edges inside a bag, which can be done in 0{k^) time. 
The most costly operation performed for each valid subset is the computation of 
the transitive closure of a graph containing 0{k) vertices. This can be achieved 
in 0{k'^^^^) time by using fast matrix multiplication ([H], [^ ) r] Restoring the 
result takes time that is linear in the size of the tree decomposition. By Lemma[2] 
the decomposition consists of 0{n) nodes. Hence, the algorithm runs in 0{n ■ 
2k . /j2.38^ time. We obtain the following result. 

Theorem 2. Given an MDP and its tree decomposition of width k of the MDP 
graph, the ASR set can be computed in 0{n ■ 2*^ • fc^'^®) time, where n is the 
number of states (vertices). 



3.2 MEC decomposition 

The algorithm is similar to the one for the ASR set in that it is also based on 
dynamic programming on a tree decomposition. Again, we assume that we have 
a nice tree decomposition with a bag of size 1 in the root. This time we obviously 
do not add the target vertex to every bag, as there is no distinguished vertex. 

As in the previous algorithm, we define a partial solution for a node d to be a 
subset of V{Gd). This subset consists of vertices that are to form a single MEC. 
A partial solution P is valid, if three conditions hold. 

1. For every v E P OVp and every edge vu G E{Gd), we have u G P. 

2. For every v E P there exists a path in P from v to some vertex in P n Bd. 

3. For every v € P there exists a path in P from some vertex in P n Bd to v. 

Note that the only difference from the algorithm for ASR set is that we have 
added the third condition. As a result we can use the dynamic programming 
scheme from the previous section, with only a slight change. When we perform 
a check that depends on the second condition (while processing a forget node), 
we need to run two symmetric checks instead of one. Let P{B' , d) denote the 
maximal partial solution for d such that P{B' , d) D Bd — B' . 

We use the following two lemmas to show the correctness of the algorithm. 
Their proofs can be obtained easily from the proofs of their analogous lemmas 
in the previous section. 



** In practice, a simple k'^ algorithm might a better choice than algebraic algorithms 
for multiplying matrices. 
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Lemma 7. For each node d and each valid subset B' C Bd, P{B',d) is a valid 
partial solution and TC{B' ,d) is computed correctly. 

Lemma 8. For every node d and MEC M such that M Bd 7^ 0, we have 
P{MnBd,d)^MnViGd). 

The difference in tfiis algoritfim is in obtaining the resuh after the dynamic 
programming step is finished. First, we find the rootmost (that is, the one closest 
to the root) node di and a vertex vi G Bd^, such that P{{vi}, di) ^ _L. In case 
of a tie, we can choose any node. We claim that Mi = P{{vi}, di) is a MEC . 
We repeat this procedure, without taking into account vertices from Mi. This 
process is continued, as long as a feasible node and vertex can be found. We now 
show that it is correct. 

Lemma 9. For each node d and v € Bd, if P{{v}, d) 7^ _L, then P{{v}, d) is an 
end-component ofG. 

Proof. From the definition of P(-, •), we have that for every u G P{{v}, d) n Vp 
and every ux € E, it holds that x € P{{v},d). Moreover, from each vertex 
of P{{v},d) there is a path to v and from v there is a path to each vertex of 
P{{v}, d). It follows that there is a path between any pair of vertices in P{{v}, d), 
so it is a strongly connected set in G, thus also an end-component. D 

This implies that our algorithm finds a collection of end-components. We 
now show that each such end-component is a MEC. Let M be an arbitrary 
MEC and let d be the rootmost node, such that Bd D M ^ 0. Since the tree 
decomposition is nice, Bd H M contains a single vertex v. From Lemma [8] it 
follows that M = P{{v},d). It is easy to see that when the algorithm picks a 
first vertex from M, it picks the vertex v defined above, and thus finds a MEC 
M. It follows easily that every MEC is eventually found by the algorithm. 

Let us now discuss the running time. As before, the dynamic programming 
step requires 0{n ■ 2*^ • k'^-^^) time. Retrieving all MECs from their implicit 
representations requires time that is bounded by the total time of building these 
representations. Moreover, the process of finding rootmost nodes requires time 
that is linear in the size of the tree decomposition. Hence, the running time is 
bounded by the time of the dynamic programming and amounts to 0{n-2^-k'^-^^). 

Theorem 3. Given an MDP and the tree decomposition of width k of the MDP 
graph, the MEC decomposition can be computed in 0{n ■ 2^ ■ k"^'^^) time, where 
n is the number of states (vertices). 

4 Static and Decremental Algorithms for MEC 
decomposition and Almost-sure Reachability 

In this section we will present the 0{m ■ k ■ logn)-time static algorithms for 
the MEC decomposition and the ASR set computation, and the decremental 
algorithms. The key would be to present two simple algorithms for the problems 
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that we will view as decremental graph algorithmic problems (decremental sec 
computation for MEC decomposition, and decremental directed reachability for 
ASR computation). We will then use dynamic graph algorithmic techniques to 
obtain the desired result. We start with the two basic algorithms. The most 
straightforward implementations of both these algorithms are not efficient, but 
we later show that they can be speeded up significantly for graphs with low 
treewidth using dynamic graph algorithmic techniques. 

4.1 Basic algorithms 

MEC decomposition. We first give an algorithm (formal description as Al- 
gorithm IT]) for computing MEC decomposition. Here, ComputeSccs denotes a 
function, which computes an array SCC that maps the vertices v into unique 
identifiers SCC[v\ of the strongly connected components in the graph. 



Algorithm 1 Mec(G) 

G' :=G 

SCC ■- ComputeSccs(G') 
3: while 3^(zVpnv{G')'^uv£E{G)SCC[u] / SCC[v] do 
4: remove u from G' 
5: see := ComputeSccs(G') 



Lemma 10. Algorithmn\is correct. 

Proof. The algorithm removes a subset of vertices of G, thus obtaining a graph 
G". It follows clearly that once the algorithm terminates, the strongly connected 
components of G' form a MEC decomposition of G' . Moreover, they are end- 
components in G (note that we use E{G) instead of E{G') in the condition in 
the third line). To show that these sets form a MEC decomposition for G (i.e., 
they are maximal with respect to inclusion), we prove that every vertex u that 
is removed does not belong to any MEC of G. If u belongs to some MEC M, 
then V must also belong to M. But, by the definition of a strongly connected 
component, u is not reachable from v, so they cannot belong to the same MEC. 
Hence, u is not contained in any MEC. D 



Almost-sure reachability. A similar algorithm to the one above can be given 
for ASR. Procedure FindReachable computes the set of vertices that are con- 
nected to s with a path in G. The formal description is given as Algorithm [2J 

Lemma 11. Algorithm[Eis correct. 

Proof. The algorithm removes a subset of vertices of G, thus obtaining a graph 
G'. It follows clearly that once the algorithm terminates, the set of vertices from 
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Algorithm 2 Asr(G, s) 



G' — G 

A ■— FindReachable(G',s) 
while 3ueVpnAJuveE(G)'" ^ A do 

remove u from G' 

A := FindReachable(G',s) 



which there is a path to s is an ASR set in G" that satisfies both global and 
local conditions. To show that it is also an ASR set in G (i.e. it is maximal 
with respect to inclusion), we prove that every vertex u that is removed cannot 
belong to the ASR set. If u belonged to the set, then v would also belong to it. 
But there is no path from w to s in G, so v cannot belong to the ASR set, and 
neither can u. D 



4.2 Static algorithms for MEC and ASR 

This section describes efficient implementations of algorithms from Section |4.1| 
that work for graphs with low treewidth. 

MEC decomposition. In order to compute MEC decomposition, we need to 
give an efficient implementation of Algorithm IT] This consists in maintaining the 
array SCG under a sequence of vertex deletions. Note that instead of removing 
vertices, we might well just remove all its incident edges. 

To maintain strongly connected components we use a data structure by 
L^cki |29j . Given the tree decomposition of a graph of width fc, it can main- 
tain the SCG array subject to edge deletions. The total running time of all 
delete operations is 0{m ■ k ■ \ogn), and every query to the array is answered 
in constant time. Thus, if fi(m) edges are deleted, the amortized time of one 
update is 0(fc • logn). 

After each update, if a strongly connected component decomposes into mul- 
tiple strongly connected components, some edges that used to be contained in a 
single strongly connected component now connect different strongly connected 
components. It is easy to see that it suffices to check the condition from the third 
line of the algorithm just for these edges. The algorithm maintaining strongly 
connected components can be easily extended to report the desired edges with 
no additional overhead. This way, we obtain an algorithm that computes the 
MEC decomposition in 0{m ■ k ■ logn) total time. 

Almost-sure reachability. We now describe an efficient implementation of 
Algorithm [2] This time it suffices to give an efficient algorithm that maintains 
the subset A C V oi vertices, such that for every r G A there exists an r-to-s 
path in G. After reversing all edges in the graph this becomes a single-source 
reachability problem. We show that by modifying the algorithm of L^cki [55], 
this can be achieved in 0{k ■ logn) amortized time. We describe the details of 
the algorithm below. 
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Decremental single-source reachability. Given a directed graph G with a desig- 
nated source s E V(G), the goal is to maintain the set of vertices reachable from 
s when the edges of G are deleted. Moreover, we assume that we are given the 
tree decomposition of G of width k. 

The algorithm is a simplified version of the algorithm for decremental all- 
pairs reachability by L^cki [29j . The description in [29^ contains an error in 
the running time analysis of the all-pairs reachability. However, the problem 
disappears, if there is only a single source. 

One of the ingredients of the algorithm is an algorithm for decremental single- 
source reachability in a DAG. The algorithm is very simple. In the beginning 
we delete all vertices that are not reachable from the source. Then, after an 
edge is deleted, we delete vertices (different from s) whose in-degree is 0, until 
all remaining vertices have positive in-degree. Note that deleting a vertex might 
decrease the in-degree of other vertices and trigger further deletions. The cor- 
rectness of the algorithm follows easily. Moreover, it can be implemented, so that 
the total running time is linear in the number of edges of the initial graph. This 
is because every edge is examined when its start vertex is deleted and this means 
that the edge itself also gets deleted. 

We can now proceed to the algorithm dealing with the general case. It main- 
tains the subgraph of the initial graph that is reachable from s. In the descrip- 
tion we treat G as a variable denoting this subgraph. To represent G we store its 
condensation Gc, that is the graph obtained from G by contracting all strongly 
connected components. It is easy to see that a condensation of an arbitrary graph 
is acyclic. Hence, we can use the algorithm given above to maintain it. On the 
other hand, to maintain the strongly connected components of G, we use the 
data structure by L^cki [53]. 

When an edge belonging to the condensation is deleted, we can simply up- 
date the condensation DAG, deleting some vertices, if necessary. All other edges 
are contained inside strongly connected components, so the deletion is handled 
by the data structure. This might cause some strongly connected component to 
break. In such case the data structure can report the condensation of the sub- 
graph obtained from breaking the component with no additional overhead. This 
subgraph is then planted in place of the appropriate vertex in the condensation. 
The details are given in [29] . 

The total running time of processing all edge deletions is 0{m ■ k ■ logn) 
and the set of reachable vertices is maintained explicitly. Also recall that for 
treewidth k we have m — 0[n ■ k). 

Theorem 4. Given an MDP and its tree decomposition of width k, the MEC 
decomposition and the ASR set can he computed in time 0{m ■ k ■ logn), where 
n is the number of states (vertices) and m is the number of edges. 



4.3 Decremental algorithms 

Both algorithms that we have described can be easily extended to decremental 
algorithms that support edge deletions. However, only deleting edges uv £ E 
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such that li G Vi is allowed. This assures that the ASR set can only shrink 
and that every end-component in the MEC decomposition is a subset of a some 
end-component from the graph before the deletion. 

Almost-sure reachability. The algorithm first runs Algorithm [2] during the 
initialization phase and computes the initial set A. The set A is maintained 
by a single-source decremental reachability algorithm. The very same high-level 
algorithm can be used to update the set A after an edge is deleted. We run 
this algorithm whenever an edge is deleted. Observe that if we detect that A 
shrinks, i.e. a subset U C A of vertices is removed from A, we need to check the 
condition in the third line only for edges that are entering this set. Thus, each 
edge is inspected at most once during the entire course of the algorithm. Hence, 
the dominating operation is the running time of the decremental single-source 
reachability algorithm, which requires 0{m ■ k ■ logn) time over all deletions or 
0{k ■ logn) amortized time for a single deletion, if /2(m) edges are deleted. The 
proof of correctness is analogous to the one in Lemma [TT] 

MEC decomposition. We use the same idea as for the decremental algorithm 
for the ASR set. In this case Algorithm [T] can be used both for the initialization 
and after an edge is deleted. By maintaining the array SCC with a data structure 
for decremental SCC maintenance, we get that the amortized time of processing 
a single update is 0{k ■ logn). 

Theorem 5. Given an MDP and its tree decomposition of width k, the MEC 
decomposition and the ASR set can be computed under the deletion of f]{m) 
player-1 edges, in amortized time 0{k ■ logn) per edge deletion, where n is the 
number of states (vertices) and m is the number of i 



Concluding remarks. In this work, we presented faster static and decremental 
algorithms for two core algorithmic problems for MDPs when the treewidth is 
low. An interesting question for future work is whether the algorithms can be 
extended to MDPs with low DAG-width (as done for parity games in [1]). 
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Appendix 

A Proof of Lemma [T] 

We prove inclusion in both directions: 

— Every vertex in A must have a path with vertices in A to s in the graph: to 
ensure almost-sure reachability, simple graph reachability must be ensured, 
and the almost-sure set should never be left. Thus the global condition is 
satisfied by A. Since from vertices outside A, almost-sure reachability cannot 
be ensured, vertices u G AnVp must have their out-going edges in A, as 
otherwise the set A is left with positive probability and from the remaining 
vertices almost-sure reachability cannot be ensured. It follows that A satisfies 
both the global and the local conditions, and since A* is the maximum such 
set we have A C A* . 

— We now argue that from every vertex in A*, almost-sure reachability to 
s is ensured. By the global condition, every vertex in A* has a path to s 
consisting of vertices in A* , and thus have an edge to a vertex in A* that is 
closer (in terms of shortest path) to s. From every vertex in A* D V\ choose 
the first edge on the shortest path (inside A*) to s. Consider the resulting 
Markov chain obtained for the set A* of vertices: then the vertex s is the 
only recurrent state and thus reached with probability 1. Hence A* C A. 

B Proof of Lemma [5] 

The proof proceeds by the induction on the depth of the subtree rooted at d. 
We consider each node type separately. The first type corresponds to the basis 
of the induction. Since we assume that P{B' , d) ^ _L and _L is propagating, we 
immediately have that all values of P(-, •) we refer to are not equal to _L. 

— Leaf The claim follows trivially. 

— Join P{B',d) is a valid partial solution, as it is a sum of two valid partial 
solutions. To show that the transitive closure is computed correctly, we show 
that every path connecting vertices of B' and going through P{B' , d) can 
be traced in TC{B',Ci) U TC{B', 02). Consider two vertices oi u,v e B' 
that are connected with a directed path p in P{B',d). Let us split p into 
subpaths by cutting it in each vertex contained in B' . Denote the result- 
ing subpaths pi, . . . , pk- We want to show that each such subpath can be 
traced in TC{B',ci) or TC{B',C2), which means that it is either contained 
inG,jP(B',Ci)]orGeJP(-B',C2)]. 

Consider a subpath pi. If it is contained within B' , it is also contained both in 
Gci [PiB', Ci)] and Gc^ [P{B', C2)]. Otherwise, consider the first vertex v out- 
side B' and w.l.o.g. assume that it belongs to G'cjP(i?',ci)]. By Lemmajsj 
any path from v to Gc2[P{B' ,02)] \ B' has to go through B' . But, by def- 
inition. Pi ends at the first vertex of B' encountered after v. Thus, pi is 
contained in G'cJP(i?',ci)]. 
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This shows that TC{B',d) C (TC(S',ci) UTC(B',C2))*. The reverse inclu- 
sion follows immediately. 
Introduce Fix B' and d. Let us first verify condition (ii) of Definition l4J 



We have that P{B' ,d) is cither P{B',c) or P{B' \ {w},c) U {w}. For each 
vertex v ^ w there exists a path from u to a vertex in B' , as it existed in 
P{B' , c). Additionally, if w G P{B' , d), the condition holds trivially for w. 
We now check whether condition (i) of Definition H holds. As P{B',c) is a 



valid partial solution, the condition can only be violated for out-edges of w. 

But w G Bd, so if the condition is not satisfied, then B' is not a valid subset 

oiBa. 

Forget In the second case, that is when P{B',d) — P{B',c), the claim 

follows easily. Let us now assume that we set P{B',d) — P{B' U {w},c). 

We only need to verify condition (ii) of Definition [4] (condition (i) follows 



trivially) . 

We know that for each v G P{B' , d) there exists a path from v to B'U{w} that 

is contained within P{B' , d). Moreover, we have checked that there exists a 

path from w to a vertex in B' . This means that for each v G P{B' , d) there 

exists a path from v to B' . 

It follows immediately that TC{B', d) is computed correctly. 

The desired result follows. D 
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